Passive Realtime Datacenter Fault Detection and Localization

Datacenters are characterized by their large scale,stringent reliability requirements, and significant appli-cation diversity. However, the realities of employinghardware with small but non-zero failure rates mean thatdatacenters are subject to significant numbers of failures,impacting the performance of the services that rely onthem. To make matters worse, these failures are not al-ways obvious; network switches and links can fail par-tially, dropping or delaying various subsets of packetswithout necessarily delivering a clear signal that they arefaulty. Thus, traditional fault detection techniques in-volving end-host or router-based statistics can fall shortin their ability to identify these errors. We describe how to expedite the process of detectingand localizing partial datacenter faults using an end-hostmethod generalizable to most datacenter applications. Inparticular, we correlate transport-layer flow metrics and network-I/O system call delay at end hosts with the paththat traffic takes through the datacenter and apply statis-tical analysis techniques to identify outliers and localizethe faulty link and/or switch(es). We evaluate our ap-proach in a production Facebook front-end datacenter.